Sequential View Synthesis with Transformer

This paper addresses the problem of novel view synthesis by means neural rendering, where we are interested in predicting at an arbitrary camera pose based on a given set input images from other viewpoints. Using known query and poses, create ordered observations that leads to target view. Thus, single is reformulated as sequential prediction task. In this paper, proposed Transformer-based Generative Query Network (T-GQN) extends neural-rendering methods adding two new concepts. First, use multi-view attention learning between context obtain multiple implicit scene representations. Second, introduce rendering decoder predict image sequence, including view, learned Finally, evaluate our model various challenging datasets demonstrate not only gives consistent predictions but also doesn’t require any retraining for finetuning.